The interim evaluation method of the national project for institutional repositories in Japan

The National Institute of Informatics (NII), Tokyo, Japan, was founded in April 2000 as an inter-university research institute. Its cyberscience infrastructure (CSI) initiative aims at providing industrial and social contributions as well as a base for international cooperation on scientific research. The initiative includes projects focusing on: networks, grid research, interuniversity public key infrastructure (UPKI), and scholarly content services. The project for institutional repositories (IRs), begun in 2004, is placed in the main focus of the CSI content project. In 2006, as the CSI initiative has put more emphasis on content services, NII began working with 57 universities to construct a next-generation scientific information resources infrastructure. One goal of the project is to spread open access repositories throughout Japan; 57 universities have joined. The other goal is to support research and development activities that facilitate the dissemination of scholarly contents. 22 R&D projects have been allocated to 37 universities. The total budget for the 2006 academic year amounts to 300 million Japanese Yen ($2.6 million USD). The project runs from April 1, 2006 to March 31, 2008. The interim evaluation of the commissioned projects was conducted from February to March 2007. The evaluators may conclude to discontinue a project, and the evaluation also focused on sharing the best practices and clarifying common problems. The check list for a proliferation project includes: the system implementation progress, the marketing practices, and the collection size. The cost for content digitization and registration was also surveyed. On the other hand, there was no uniform checklist for an R&D project, since the range of these projects varies. Five committee members are in charge of evaluating all 22 R&D projects and discussing the direction of R&D, including the possibility of restructuring commissioned projects. The paper describes the methods and results of the interim evaluation. Introduction and background The National Institute of Informatics (NII) was founded in April 2000 in downtown Tokyo as an inter-university research institute. It not only conducts comprehensive research on informatics, but develops and provides an advanced infrastructure for disseminating scholarly information. Its cyberscience infrastructure (CSI) initiative aims at providing industrial and social contributions as well as international cooperation on scientific research. The initiative includes the following projects: – SINET. NII provides a 100M network infrastructure shared among the academic institutions in Japan. – NAREGI. NII promotes a research grid middleware to enhance supercomputing in scientific research. – UPKI. NII collaborates with universities to construct an interuniversity public key infrastructure. – Next-generation content services. Infrastructures for scholarly information resources are under construction on the basis of existing scholarly content services that were established by collaboration between NII and universities. Thus, NII serves as an academic infrastructure for all Japanese universities as well as international institutes. There were 726 universities in Japan (87 national universities, 86 municipal universities, 553 private universities, and 4 private correspondence education colleges) in the academic year 2005. Their backgrounds were diverse and the network and library systems are not uniform. Nevertheless, NII stepped forward to serve many universities as quickly as possible. 2004-2005: Pilot projects Interest in institutional repositories in Japan arose around 2002. The report in 2002 by the Subdivision on Science the Council for Science and Technology at the Ministry of Education, Culture, Sport, Science and Technology (MEXT) of Japan, aiming at improvement of academic information dissemination infrastructure, asserted importance of online information dissemination from universities, though not explicitly mentioned IR. Chiba University began research to develop a prototype system in 2002. In 2003, the MEXT published a report that emphasized the role of university libraries in the dissemination of academic information, in particular in the areas of humanities and social science. Since then, the idea of an institutional repository has formed. In 2004, NII started an experimental project with 6 universities to work on software implementation and to conduct trials on Dspace and eprints. IR was placed in the CSI initiative as an indispensable component of the next-generation content services in 2005. Nineteen universities have worked together with NII in a pilot project under the CSI framework to propel IRs. On June 28, 2005, the MEXT Council published an interim report on university libraries to explicitly state the significance of institutional repositories in the dissemination of academic information resources. By June 28, 2006, 17 institutional repositories were running; these repositories hold a total of 62,423 items. 2006-2008: Institutional repositories as content infrastructure The Subdivision on Science in MEXT published an interim report on March 23, 2006, which explicitly emphasized importance of IR. A two-year full-fledged initiative for developing a next-generation scientific information resources infrastructure has been initiated and will run from April 2006 to March 2008. A content board in NII consisting of several deans from university libraries was institutionalized on September 13, 2005. A working group was installed to discuss the operation of the initiative. The center of scholarly information resources, established on August 3, 2006, also provides support to the initiative. From 2006-2007, 57 universities will be selected from the 77 applications that were received to join a full-scaled project to promote open access repositories for scholarly communication; 40 IRs were running in Japan as of April 2007. Total 37 universities among the 57 partners carried out 22 research and development projects of a wide range on smooth operations of IRs. They include: Connections to link resolvers; Integrated searches; Metadata for multi-type data; Connections to faculty performance databases; IR community developments; Copyright policies of Japanese academic journals; XooNIps library modules; E-publications in IRs; and Development of IR evaluation methods. The 2006 budget was 300 million Japanese Yen ($2.6 million USD). Interim evaluation of the project The interim evaluation was conducted from February to March 2007. The partner universities submitted a progress report and a proposal for the 2007-2008 academic year (from April 1, 2007 to March 31, 2008). The content working group members, 13 members in 2006-2007, were assigned to one of the two evaluation committees: one for the promotion projects, the other for the R&D projects. For both project classifications, three committee members will evaluate a project using the following four rankings: Excellent (4); outstanding practice, deserving a chance to present the progress at the debriefing session Good (3): progress as planned. Fair (2): progress behind the initial plan; the initial target can be reached. Poor (1): progress delayed; need to discuss possibility of cessation. Development of measures Measures for the assessment of IR activities were developed by a commissioned R&D project conducted by Chiba and Mie Universities. The measures primarily focus on the inputs and process. Output measures were proposed but not implemented at this time. Instead, Chiba University is now analyzing the log files of the IRs in partner universities to establishing a standard for output measurement. There are four aspects for the evaluations: system, institutionalization, marketing activities, and input. The first two areas were evaluated to check the progress and project scheduling, while the latter two tried to assess the efficiency and effectiveness of operations with the consideration of future combination with the output measures. Results: Expansion projects All the universities have made reasonable progress. Some are still on the way towards system installment, but their IRs are expected by the end of the project term.